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ABSTRACT 

- The higher education community needs_measures of the 

value addid to student development by the college experience. The 
American College Testing Program (ACT) provides a quick, easy method 
for estimating the eKteiit of student growth in general education* An 
institution can test seniors With the ACT College Outcome Measures 
Project (COMP) exam^ then subtract from the senior score an estimated 
freshman score obtained from a_*'cbncbrdance table** that is based on 
the known relationship (r = .70j_between freshman ACT Assessment 
Composite score and f rishmanCOHP total score. Studies using scores 
for 4,200 seniors and_2,i00 freshmen tested during a two-year period 
at the University of Tennessee, Knoxville (UTK) indicate that this 
method is not sufficiently reliable or valid to serve as the basis 
for making precise judgments about the relative quality of general 
education programs at various institutions, as at least one state 
coordinating agency for higher education has attempted to do. These 
studies show that estimates of student score gain on the COMP exam 
may be cJe rived from systematically biased samples if not all students 
have ACT Assessment scores , and can be in error by as much as 66 
percent^ Moreover^ estimated score gain bearsa negative relationship 
to a number of institutional variables generally associated with good 
practicein higher education. Appendices contain contingent 
proportions for students with and without ACT scores on selected 
variables (1984-85 and 1985-86 senior sallies) and mean gain 
scores for selected variables (same periods and same sample) ^ 
(Author/JAZ) 
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ABSTRACT 



The higher education community needs measures of the value added to 
student development by the college experience. ACT provides a quick, easy 
method for estimating the extent of student growth in general education. An 
institution can test seniors with the ACT College Outcome Measures Project 
(eOMP) exam, then subtract from the senior score an estimated freshman score 
obtained from a "concordance table" that is based on the known, relationship 
(r = .70) between freshman ACT Assessment Cdtnpdsite score and freshman COMP 
Total score. Studies using scores for 4200 seniors and 2100 freshmen tested 
during a two-year period at the University of Tenhesseej Krioxville indicate 
that this method is not sufficiently reliable or valid to serve as the basis 
for making precise judgments about the relative quality of general education 
programs at various institutions, as at least brie istate codrdiriatirig agency 
for higher educatiori has attempted to do. These studies show that estimates 
of studerit score gain on the COMP exam may be derived from systematically 
biased samples if not all students have ACT Assessment scores, and can be in 
error by as much as 60 percent. Moreover , estimated score gain bears a 
negative relationship to a number of institutional variables generally 
associated with good practice in higher education* 
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ESTIMATED STUDENT SCORL! GAIN ON THE ACT COMP EXAM: 
VALID TOOL FOR INSTiTUTiONAL ASSESSMENT? 



Trudy l-J . Banta^ E. Warren Lambert, Gary R. Pike* 
James L. Schmidhammer ^ Janet A. Schneider 

Purpose 



The College Outcome Measures Project (COMP) exam of the American College 

Testing Program (act) has been available for use by colleges and universities 
since the academic year 1979-80. The exam was designed ''to measure knowledge 
and skills relevant to successful functioning in adult society" (Forrest, 
1982, p. 11). The COMP exam has been administered at least once on some 350 
campuses, and it is used annually by approximately 100 four-year institutions 
for the purposes of assessing and improving their general education programs. 
Despite this rather substantial base of institutional experience in using the 
COMP exam^ surprisingly few studies have appeared in the literature concerning 
its technical qualities. The_ purpose of this paper is to provide some 
evidence bearing on the reliability and validity of estimated score gain on 
the COMP, a topic previously unexplored in the literature. 



The COMP exam provides a total score and six subscores, three in content 
areas and three in process areas, as follows: 



Cootent Areas 

Functioning within Social Institutions 
Using Science and Technology 
Using the Arts 

Process Areas 

Coinmuhicat ing 
' Solving Problems 

Clarifying Values 

The exam is available in two forms: the Composite Examination, which 
contains a multiple-choice section as well as additional exercises that permit 
students to construct their own written answers arid to record a speech; and 
the Objective Test , consisting of 60 multiple-choice items each of which has 
two correct responses. The correlation between the Total scores on the 
Composite and Objective forms is approximately .80. Most of the institutions 
using the COMP exam on a continuing basis employ the Objective Test berausr. it 
takes less time for students to complete and is less expensive to administer 
and score. 

Review of Literature 

Persistent attempts to locate in the ERIC database institutional studies 
using the COM? exam have yielded no more than a dozen citations over the years 
since the instrument was first marketed. Most of the studies are exploratory 



in nature, describing the results of an institution's first administration of 
the instrument to a sample of istudentis; 

For example^ Jones (1982) reported the outcomes of teisting all (93) 
entering_f reshmen at Nazareth College in 1980, and a sample of 25 senior 
volunteers in Spring_i98i, using_the COMP Objective Test; The freshman and 
senior groups were found to be equivalent in ability. as measured by the ACT 
Assessment, and the senior mean Total score was 35 points higher than the 
freshman mean. The conclusion was drawn that for freshmen who remain at 
Nazareth College for four years ^ significant growth can be expected in the 
areajs of knowledge and skill assesseid by the COMP exam. 

Ward and Prihgle (1981)_ tested a total of 99 graduates of hoh-tradit ional 
pbstsecbhdary programs in Illinois (a "university without walls" and other 
individualized programs) and concluded that the COMP Objective Test was not 
biased against nontraditional students. In fact, the nontradit ional sample 

Communicating and Using Science and Technology 
subscaies than did a norm group of traditional students from 30 other 
inst itut ions . 

Dumont and Troelstrup (1980) attempted to relate COMP scores for a random 
sample of 112 seniors at Tennessee Technological University to a series of 
self-reported ratings of progress toward achieving a set of institution-wide 
goals for general education. The low correlations obtained indicated that the 
less expensive self-report data could not be used as a substitute for the COMP 
exam scores. 

Schomberg et al . (1981) also failed to find meaningful relationships 
between COMP exam scores and self-reports concerning perceived benefits of 
college and satisfaction with levels of skill and knowledge. The authors' 
?^P?^^^^^^ ^^sting 96 graduating seniors at the University of Minnesota led 
them to conclude that the COMP exam is too easy to differentiate among 
students of high ability. 

In a study involving 696 students in an urban community college setting, 
Kitabchi (1985) found that entering ability (ACT Assessment Composite) was the 
most important predictor of success on the COMP_exam. Significant 
relationships also were found with age and racial-ethnic group; older students 
and white students outscored younger students and blacks. 

Student Score Gain 

Many colleges and universities are drawn to the COMP exam because it 
offers the promise of providing object ive evidence of student growth in 
generic knowledge and skills — "value added" — over the years of association 
with an institution. One of the developers of the exam has written^ 

"_. . .we - strongly recommend that an institution gather and use 
empirical evidence of the degree to which its genj2ral 
education_program is providing intended benefits... We 
suggest that the empirical evidence include. . . Student test 
score gains from entering freshman status to graduation" 
(Forrest, 1982, p. 4). 
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Since a hew form of the CdMP exam is developed each year, stadents who 
persist at an institution can be tested upon entrance and again at the end of 
two or four years of cbl lege .experience with equivalent forms of the same 
exam. In reality^ however^ few_ institutions want to wait even two years to 
measure the_growth their general ec' icaticn program is producing in continuing 
students. In addition^ many institutions are jnwil.ling to. invest the 
resources required to test entering students > and thus administer the GOMP 
exam to graduating students only. 

Partially in response to this impatience on the part of institutions ^ the 
developers of tue COMP exatn have offered to provide ah estimate of student 
gain in Total score. Using evidence that the correlation between the Total 
score achieved by freshmen on the Objective Test arid their ACT Assessment 
Composite score is about .70 (Forrest and Steele, 1932, p. 57), the developers 
have constructed a concordance table (see Table 1 for the 1986 version) from 
which institutions may estimate a mean freshman COMP exam score if-they have 
the mean ACT Assessment Composite score for a group of gradua t ing^sr udeot s who 
have actually taken the COMP, By subtracting the estimated freshman score 
from the actual score for graduating students, an estimate of score gain, or 
value-added^ can be obtained,. The ACT staff provide no estimates of gain on 
the six COMP subscores^ and they caution in the introduction to the 
cbricbrdance table that it is "not appropriate to use to estimate individual 
student growth." 



The_developers of the COMP have riot published the methodology for 
constructing their concordance table or its revisidris, nor any evidence of the 
reliability or validity of estimated score gain. The review of literature 
summarized in the f9regoing section of this paper reveals no studies of the 
reliability or validity of estimated gain, 

ACT offers the service of estimating student score gain to institutions 
using the COMP exam. Many of the more than lOO four year institutions using 
the COMP annually consider the estimate of gain in assessing the relative 
effectiveness of their general education program vis-^^vis those of peer 
institutions arid/or institutions in the national norm group. One state 
coordinating agency^ the Tennessee .Higher Education Commission, uses the value 
of estimated score gain as pne_of five criteria. for awarding an annual 
financial supplement to each of the state's public colleges and universities, 

Estimated score gain is being employed iricreasirigly to make decisions 
that can_have a far-reach ing impact dri iristitutidns df higher education in the 
United States, is the importance currently attached td this cdristruct 
justifiable?^ Can it be used by institutions to identify strengths and 
weaknessesof their general education programs and to suggest directidns fdr 
improving such Programs? As more and more states take action td encourage 
public postsecondary institutions to undertake programs designed to assess 
student outcomes, can state officials regard estimated score gain as a tool of 
sufficient reliability and validity to serve as the basis for making precise 
decisions concerning the relative quality of the general education programc at 
various institutions? 

The University of Tennesseei Knoxville ^ (UTK) has the most extensive COMP 
database in the country. Since 1980 several hundred seniors at UTK have been 
tested annually using the COMP Objective Test, In 1985 the test became a 
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Table I 

1986 Revised Cbricb.rdance table 
of ACT Comi5dsite Scores and CDblP Objective Test 

Tocal Score EquivalehC 

(Based on 13,552 Entering Freshmen) 



Notie: It is appropriate to use this table only for estimating the 
COMP Mean Tot ^.1 Score that a sample of Sophomores or Seniors might 
have obtained had chey talcen_the COMP Objective Test or Composite 
Examination instead of the ACT Assessment as High school seniors or 
entering college freshmen. This table is not appropriate to use to 
estimate individual student growth, 

ACT-Composite Scores Equivalent COMP Total Sco^res 
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graduation requirement for every senior^ and during 1985-86 it was 
administered to oyer 3^000 seniors, Since 1983 at least 600 freshmen havt; 
been tested annually at the beginning of their first, quarter at UTK. In 1985, 
ahdagairi in 1986^ apprpximately 1 ^800 freshmen^ or half of the , first-time 
full-time freshment took t^e COMP exam. UTK is in a unique position to 
provide eyidencs of the_techhical quality of estimated student score gain on 
the ACT COMP Objective Test. 

Prdblemis with Estimated Score Gain 

For three j^ears systematic studies have been conducted using the COMP 
database at the University of Tennessee, Knoxvilte- The following problems 
witti estimated gain have been discovered, confirmed, and reconfirmed on 
successive data sets. 

1. Estimated gain has a large standard deviation. 

2. The regression equations used to estimate freshman COMP _ 

scores from entering ACT Assessment scores differ significantly from 
one student cohort to another. Thus the ACT staff must re-calibrate 
its cbncbrdsince table periodically and issue reyisions. Actual 
testing of UTK freshmen has revealed that use of these concordance 
tables results iri a significant over-estimate of the mean UTK freshman 
COMP score, and thus a significant Urider-estimate of gain at UTK. 

3. Estimated §3^" is 2 ^^^^Ifscore — the difference between 
scores on two tests of imperfect reliability ~ and thus is less 
reliable than either of the tests u^ed in its computation. 

4. A significant proportion of graduating seniors has no ACT 
Assessment Composite score upon which to base an estimate of freshman 
COMP score. Moreover, seniors without entering ACT scores differ in 
systematic ways from those who have such scores. 

5. The validity of estimated score gain as an indicator of 

the effectiveness of ah institution's general education program is 
called into serious question bh the basis of a series of 
counter-intuitive relationships found to exist bietweeh estimated gain 
and certain demographic and institutional variables. 

An explanation of each of these findings appears in the paragraphs that 
follow. 

Standard Deviation of Estimated Gain 

The estimate of COMP score gain based on the freshman \CT Assessment 
Composite score has a large standard deviation. The data in Table 2 provide 
evidence of the variability of ACT Assessment Composite scores, senior COMP 
Total scores and estimated COMP score gain for UtK seniors who took the COMP 
in 1985* and in 1986*. While the mean estimated gain is less than half the. 
value of the ACT scpre^ its standard deyiatidn is almost tliree times that of 
ACT scores. The value of the standard deviation bf estiirated gain for 1986 is 
60 percent greater than the yalue of the mean itself; and the 95 percent 
confidence interval for the 1985 mean is almost 20 percent of the value of the 
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meani thus indicating that in a given year the level of estimated gain for UTK 
could be in error by as much as 20 percent. 

Table 2. Variability of ACT Assesstnient Composite Score, 
Senior _eOMP Total Score, and Estiimated COMP Total Score Gain 
for UTK Seniors taking C^P in 19855% and 1986*. 

Standard 

Variable Mean Deviation 

ACT Score 

1985 902 21.26 4.89 

1986 2259 21.81 4.73 

Senior COMP Total Score 

1985 1284 187.91 15.99 

1986 3276 188.81 16.08 

Estimated Gain 

1985 843 9.16 12.81 

1986 2226 8.81 14.27 



*Fdr convenience the samples are labeled 1985 and 1986 in the narrative. 
Actually, the 1985 sample includes seniors tested daring the four quarters 
Fall 1984 and Winter, Spring, ahd_Summer 1985. Likewise, the 1986 sample 
includes students tested between Fall Quarter 1985 and Summer Quarter 1986. 

Stability of the ACT Concordance- tables 

J^^^^^^^?^^ ;^^?L^s^™^ti^^ of-gain in Total score on the COMP exam, 

(gain on the six subscales_is not estimated by ACT) its developers 
periodically provide institutional users with a concordance table that pairs 
ACT Assessment Composite scores with COMP Total scores (see Table 1). The 
concordance tables which may be used to estimate a freshman COMP Total score 
for anv student with an ACT Asse:;sment Score, was first issued in 1983, then 
revised and reissued In 1983, I9i5y and 1986. 

In the absence of a complete description of the methodology for 
constructing the concordance table^ the best guess about its derivation is 
that bivariate regression was used to relate freshman ACT Assessment Composite 
scores to COMP Total scores for a sample of freshmen who took the COMP exam, 
"^^^".s.^l-^^^^^ P°i*^ts from the least squares regression line relating the two 
tests were used in the concordance table. 

Employing this technique, the regression equation for the "1983 Revised 
Concordance Table" was found to be (3.13837 * ACT score) + 113.31038_. This 
equation rjunamarizes the concordance table quite adequately: a correlation 
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coefficient of ,998 was calculated between the best fit equation and the 
concordance table. 

To datie there _are three revisions of the original concordance table, each 
described by a different regression equation (see Figure 1), For the 1935 
revision, the formula_is (3,11746 * ACT score) + 113.36025, and for the 1986 
revision it is (3.36573 * ACT score). + 107.37379. Not surprisingly , the three 
regression tines, based on three different student samples , are significantly 
different from one another. This instability of the concordance table from 
year_to year_compl icates greatly the process of estimating score gain for 
seniors, since the estimated freshman COMP score for seniors having taken the 
ACT Assessment in different years must be derived from several concordance 
tables . 

Inan effort to test the_accuracy of the ACT concordance tables in 
estimating freshman COMP scores at UTK, freshman score estimates derived froi 
the appropriate ACT Revised eoncordance_Tables were compared with actual 
freshman scores for 1637 freshmen who took the COMP in 1984 and 1915"; A 
t-test for paired measures revealed a significant difference (t = 18.46; 
p<.0001) between the estimated and actual freshman COMP score7. In fact the 
estimate of the average freshman Total score derived from the ACT concordance 
tabte_was 6 points higher than the actual average Total score achieved by UTK 
freshmen. Since score gain is figured by the ACT staff by subtracting the 
estimated average freshman COMP Total score from the actual average COMP Total 
Score attained by seniors, this finding means that the ACT estimate of student 
score gam for UTK seniors ~ which, for the years under consideration was 10 
was 6 points lower than it should have been. 

_While these results_need to be confirmed by studies at other 
institutions, it is worthy of emphasis that use of the ACT concordance tables 
produces estimates of score gain on atleast brie campus that are substantially 
different f rom_ est imates based on actjjal freshmari COMP scores at that 
institution. In fact, the ACT estimate of score gain for UTK seniors in 1985 
ha ve^b ee n ''^^ ^'^'^^'^ tjy 60 percent — iz^^m-^^ent lower than it should 

- - ^^g^^ssiori analysis was used to describe more accurately the relationship 
be tween_ freshmari ACT Assessment scores and actual_eOMP Total scores for UTK 
freshmen. The resulting regression equation for estimating COMP Total from 
ACT Composite was (2.64705 * ACT Score) + 117.61108. This equation explained 
51 percent of _the_variarice in actual freshman COMP scores. Examination of the 
variance not explained by the equation indicated that the residuals were 
randomly distributed about a mean of 0.00. 

The line described by this equation is plotted in Figure 1 with the three 
lines that. define the points on the ACT coricbrdance tables^ making the 
statistically significant differences visually apparent. Figure 1 also 
illustrates the fact that high ability students were most disadvantaged by the 
application_of the ACT estimate. For freshmen with ACT scores of 2S or 
higher, COMP scores estimated by the ACT staff were 10 points higher than the 
actual COMP scores achieved. As a result, ACT underest imaged gain by 10 
points for this group. 
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FIGURE 1 

REGRESSION USES FOR CONCORDANCE TABLES 




ACT 1983 Revised Concordance Table 1985 Revised Concordance Table 
UTK Concordance Table Based on Freshmen Tested in 1984 and 1985 



0 = 1986 Re^is^d concordance Table 



Reliability of Estimated Gain 



Isaac and Michael (1981) have defined reliability as the reproducibility 
(repeatability) of a measure, the internal consistency of a measure ^ or the 
extent_to which a measure representis a "true" test score. According to these 
authors, an_ important property of the reliability coefficient is that it 
indicates_rhe extent to which variability in a score represents true score 
variance rather than error^ For example, if the reliability coefficient for a 
given instrument is -80, this means that 80 percent of the variation in scores 
on this instrument is attr ibutable_ to true score variance. The remaining 20 
percent of the variation may be attributed to error. 

Obviously, efforts to assess student outcomes of higher education must be 
based on reliable indicators^ particularly when resource allocation decisions 
are based bh the results of assessment as they are in Tennessee. Studies 
carried cut at the University of Tennessee ^ Knoxvi lie have confirmed the 
conclusion of the test developers that the internal consistency of the COMP 
Objective Test is adequate for use in program evaluation., Forrest and Steele 
(I982,_p,57) have reported a Crdribach alpha estimate of .84^ and the UTK 
estimate of alpha reliability is .76. The estimate of internal consistency 
for the ACT Assessment is :;85 (ACT, 1973). 

j^?^^?^?? gain on the COMP is based on the score 

obtained from two instruments of acceptable reliability, coticnon setise would 
suggest that estimated score gain also woald_have acceptable reliability. In 
reality however, estimated gain is a difference score, and has much lower 
reliability than might be assigned. When scores from correlated measures are 
subtracted from each bther^ the resulting difference includes more of the 
unreliability i or error variance, and less of the true score variance than 
does either score taken individually (Ferguson, 1981). 

Ferguson has proposed the following equation to assess the reliability of 
a difference score: 

^dd= (^xx ^ ^yy" ) / (2 - 2t^^) 

Where : 

r = the reliability of the difference score (estimated gain); 

^xx"^ the reliability of the initial measure (ACT Assessment score, and 
thus estimated freshman COMP score); 

r yy = the reliability of the subsequent measure (senior COMP); and 

r xy = the correlation between the initial and subsequent measure. 



This equatidri has been used to calculate the reliability of estimated gain 
first using data reported by the ACT developers and then data derived from 
testing seniors at UTK. 



have reported a correlation coefficient of .60 between 
ACT Assessment Composite scores and COMP Objective Test Total scores; for 257 
seniors (1982, p. 57). Since the estimated freshman COMP score is simply a 
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transfbnnatibn_bf the ACT Assessment score (see the discussion of the 
construction of the cohcordahce tables in the preceding section), an estimate 
of the reliability of estimated gain can be calculated as follows using 
Ferguson's equation. 

£ = .as -H-^SA - (2 * .60) 
2 - (2 * .60) 



4^ = 



;61 



UTK studies using senior and freshman COMP scores for 1985 yield a 
reliability coefficient_of .76 for the senior_CbMP score and .65 as the 
intercorrelation between freshman and senior scores. Using these figures in 
Ferguson's equation, the reliability of estimated gain is .44. 

_ Using UTK data for 1986, the reliability of the senior GOMP score is .76, 
and the intercorrelation between freshman and senior scores is .58. These 
figures yield an estimate of reliability of .54. 

With either of the UTK estimates of the reliability of estimated gain, 
about half of the gain score is due to error rather than true score . This 
finding is one more piece of evidence that the level of dependabilityof 
estimated gain is simply too low to serve as the basis for making decisions 
about program quality or the allocation of resources. 

Missing ACT Assessment Scores 

About a third of the students achieving senior status at UTK have no ACT 
Assessment Composite score to use in calculating an estimate of score gain on 
the COMP exam. The data in Table 3 illustrate the fact that in the years 
since the COMP exam has been given at UTK, the annual percentage of the senior 
sample without ACT Assessment scores has ranged from a low of 17 to a high of 
45, with an average of 32. 

Table 3. -^izes of Samples of University of Tennessee Seniors 
Taking the COMP Exam and Having No ACT Composite Score -1980-1986 



1980 1981 1982 1983 1984 1985 1986 

# Seniors 

Taking COMP 165 680 644 700 851 1028 3195 



1980-86* 
Average 



it w/o ACT 
Composite 



46 



NA 



177 



318 



148 



357 



1023 



Percentage 
w/o ACT 
Composite 



28% 



NA 



27% 



45% 



17% 



35% 



32% 



32% 



*FigUres based on reports compiled by ACT; summer test scores not included. 

The problem of havitig a large prdportibn of students v/ithbut an ACT 

Assessment score to use in estimating gain on the COMP is not unique to UTK. 



EKLC 



IK 



i I 



In ihahy iristitut ions that, use academic aptitude test scores in making 
adinissibris decisions at the freshman levels there are interhatibhal and non- 
traditibrial students for whoih the requirement for submitting these scores is 
waived. Mbriebver, students whb transfer tb the institution with at least 
sbphbtnore statuis ihay be admitted primarily bh the basis of their academic 
performance at other institutions, and brice again the requirement that they 
submit evidence of entry- level ability is waived . 

The fact that qualitative decisions about an iristitut ibri ' s general 
education curriculum may be made on the basis of a calculation that excludes 
one-third of the population experiencing that curriculum is sufficiently 
troublesome to raise doubts about the validity of estimated score gain. But 
if one could demonstrate that the students excluded did not differ iri any 
important ways from the included group, some of the doubt could be dispelled. 
In point of fact^ at UTK there are systematic differences between seniors who 
have ACT Assessment scores and those who do not. These differences are 
illustrated in Table 4 and Appendix A. 



UTK senibrs in 1985 and 1986 were divided into two groups those having 
ACT Assessment scores and those without them. The two groups were compared on 
some 50 variables, including dembgraphic characteristics^ participation in 
campus activities, and satisfaction with experience at the University. 
??P?F?^^_^^^"^^^^^^ analyses were conducted fbr each year, and a rather 
conservative (^01) level of significance was employed due to the large sampile 
sizes ^l>38i in 1985 and 3,520 in 1986) and the large number of statistical 
tests performed. 

_ Even if the additional restriction of using only the variables found to 
differentiate those with and without ACT scores in both years is applied, 
there are clear distinctions between Che two groups on such fundamental 
characteristics as age^ racial-ethnic group, parents' level of education^ and * 
parents' income. UTK seniors without ACT scores in 1985 and 1986 were older 
and mbre likely tb be blacky come _ from lower-income families, and have parents 
pbssiessihg less than a high school education. 

Seniors without entering ACT Assessment scbres also had lower high school 
grades and were less likely to receive an academic schblarship fbr college, 
"^^^y transferred tb UTK from another institution. 

They tended to_live of f _campus and not to participate as frequently as 
students having ACT scores in such campus_activities as intramural spbrts, the 
film series, concerts, and plays. In keeping with their limited level of 
involvement in c^pus life, seniors without ACT scores were less satisfied 
with their social experience at UTK, and less satisfied with their overall 
college experience than were their more-involved peers possessing ACT scores. 

Estimating the impact on overall COMP score gain of eliminating student 
without ACT_ Assessment scbres is problematic. The demographic indicators 
coupled with the Ibwer high school _GPA suggest that this third of the senior 
class wbuld have a lower average ACT Composite score if this measure of their 
academic aptitude were available. At least at UTK^ students with lower 
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TABLE 4 i2 
Chi-Square Results for Differences Between Students 
With ACT Scores and Stiiidents Without ACT Scores 
oh Selected Variables 
(1985 and 1986 Seniors) 







1984-85 


4^5-86 


Qnest fon 




df 


X2 


Student's Race 


1 


8.235* 


■j 


293.108** 


Gender 


1 


2.631 


1 


3.722 


Marital Status 


1 


1.174 


I 


4;019 


Transfer Students 


1 


348.806** 


I 


469.832** 


High School GPA 


1 


17.701** 


i 


32.974** 


College GPA 


1 


0.171 


i 


1.196 


Type of Cpmmuhity 


4 


10.073 


4 


10.350 


Mother's Education 


5 


45.490** 


5 


98 .509** 


Father's Education 


5 


53.226** 


5 


109.057** 


Parents' Ihcbme 


8 


49.594** 


8 


95.682** 


Received Academic Scholarship 


1 


29.615** 


1 


46.709** 


Received Grant or Loan 


1 


0.321 


1 


0.383 


Where the Student Lives 


4 


22.658** 


4 


55.424** 


Where the Student Works 


2 


1.527 


2 


18.823** 


Hours per Week — Working 


4 


7.192 


5 


77.733** 


Hours per Week — Studying 


5 


3.22C 


5 


5.201 


Hours per Week — Library 


5 


10.532 


5 


11.170 


Hours per Week — Pleasure Reading 


5 


7.97 


5 


3.986 


Hours per Week — Television 


5 


5.852 


5 


12.450 


Hours per Week ~ Social Activities 


5 


35.032** 






Hours perWeek — Class 






5 


38.320** 


Commuter Student 


2 


23.371** 






Participate in Freshman Orientation 






2 


123.704** 


Member Club or Professional Org. 


1 


2.121 


I 


28.591** 


Part ic ipate Internship 


1 


2.988 


I 


4.643 


Participate Co-op Program 


1 


1.020 


I 


0.611 


Participate University Studies 


1 


0.095 


I 


3.015 


Participate College Scholars 


1 


1.623 




0.740 


Participate Honors English 


1 


24 .099** 




33.908** 


Participate Honors Math 


1 


1.755 




2.614 


Campus Plays Attended 


4 


57.127** 


4 


81.143** 


Campus Films Attended 


4 


39.361** 


4 


93.391** 


Campus Concerts Attended 


4 


52.807** 


4 


87.323** 


Used Career Planning & Placement 


1 


9.074 


1 


32.299** 


Participate intramural Gports 


3 


55.946** 


3 


87.669** 


Used Student Counseling Center 


3 


0-903 


3 


14.557* 


Used Student Employment Center 


3 


2 408 


3 


2.398 


Used Computer Center 


-J 


1 910 






Clojse Rip 1 at" 1 hh 1 h Wi "P^rnl f"v 




fy fill. 


'5 


C Q T ^ 




o 






CO £ 9 1 


rULCXgLl ijclIlgUclgc LiUUlScWOFK 


9 


A A c; 7 


z 


7 . OUo _ 


LNciLuircii. ocj.ence uouf seworK. 


A 
H 




r 


90. jo4'*'* 


iria uiicma L. o uoursewor^K. 


H 


Q A A 0 




Ipp • -?4p'^ 


numan X L. i.co LfOursewurK. 


6 


4.890 


0 






6 


13.941 


6 


168.239** 


History Coursework 


3 


49.536** 


3 


5.903 


Satisfaction with Social Experience 


3 


18.232** 


3 


30.779** 


Satisfaction with Academic Experience 


3 


13.105* 


3 


5.603 


Satisfaction with Overall Experience 


3 


12.288* 


3 


15.703** 


Expectation of Overall Experience 


2 


3.644 


2 


2.288 


Offered Job Upon Graduation 


2 


5.281 


2 


2.977 


Admitted Graduate Professional School 


2 


5.409 


2 


2.492 


Plan to Remain in Tennessee 


2 


0.694 


2 


12.430* 



A detailed presentation of significant results is cdritairied in Appendix A. 



ERIC 



entering ACT scores stand to benefit more than those with high scores from 
their college experience as evidenced by gain in COMP scores (see Table 5). 
On the. other hand, students who are highly involved in the campus experience 
generally are assumed to profit more from it {Nat ionat Institute of Education! 
1984), and students without ACT Assessment scores appearto be far less 
invested in campus activities than their peers who have these scores; Th'3 
only way to tell whether the COJIP exam scores of students without freshman ACT 
scores will have a negative or positive (or neutral) imp/--ct on overall campus 
gain bri the COMP exam is to give the COMP to samples of freshmen with and 
without ACT scores^ then wait until these . students become seniors , administer 
the COMP again, and compare their respective levels of score gain (or loss). 

Relations hips of Selected V ariables to Estimated Score Gain 



ACT staff have not published any information about the relationship 
between ability and estimated score gain- on the COMP exam. Inspection of the 
"COMP Gain" column in Table 5 provokes speculation that there is an inverse 
relationship between ability, as measured by the ACT Assessment Composite 
scores and estimated score gain on the COMP exam. Studies conducted at UTK 
have yielded a Pearson correlation of -.44 between ACT Composite and estimated 
gain. 

These data indicate that students with the lowest entering ACT scores 
have the best chance of achieving, high gain scores on the COMP exam as 
seniors. Because the test has a low ceiling (Schomberg et al., 1981), 
freshmen with ACT Assessment scores of 28 or higher routinely score 2013 or 
^9^^ possible 240 points bh the exam, and thus have little chance of 

achieving large score gains when they take the COMP a second time, as seniors. 
In fact, due to the_statistical phenomenon of regress ibn toward the mean^ 
students with_Aef Assessment scores above 30 who achieve very high scores on 
the COMP as freshmen are likely to make O^wet scores when they take the COMP 
again as seniors. 

No studies linking student experiences to actual COMP score gain have 
appeared in the literature. Since few, if any, four-year institutions have 
substantial numbers of students who have taken the COMP Objective Test as 
freshmen and again as .seniors, little is known about longitudinal gain on the 
COMP. Freshmen were first tested at UTK in Fall 1983; thus the first studies 
involving actual score gain at this institution are not likely to be done for 
another year. 

Most institutions administer the COMP exam with the hope that students' 
^^9^^? P^^vide some iridicatibh bf the success of the general education 

^°?F^^°_^°?>_^?^ ?^8B?^^ directions ior change_that may improve the program. 
^^99^ ^.^ ?"*P?y ^f™o*^strated that COMP exam scores are largely_a 

function of student ability (Kitabchi, 1985 and Forrest arid Steele, 1982), 
many institutions have focused_attention on score gain as the better indicator 
of the effectiveness of the_general education program. As institutions 
consider the estimates of score gain provided by ACT, it would be very helpful 
if _ they could obtain some suggestions for actions that might serve to increase 
this gain. 

However^ research using the UTK data on score gain does not provide the 
kind of direction for program improvement that most institutions would want to 
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Table 5 
Senior COMP Mean^ 
Estimated Gain Mean ^ and 
Standard Deviation of Gain 
by ACT Composite Score 
for 1985 and 1986 UTK Seniors 



ACT 




COMP 


GAIN 


Score 


N 


Mean 


Mean 


9 


13 


166.69 


27.31 


.10 


2i 


167.19 


25.43 


11 


32 


173;44 


27.56 


12 


42 


168.52 


19.24 


13 


54 • 


175.52 


23.63 


14 


84 


173.99 


I9;02 


15 


91 


177.70 


19.20 


16 


i 15 


178.44 


16.94 


17 


159 


181 .35 


!6.05 


18 


173 


182.69 


14.42 


19 


188 


183.97 


12.32 


20 


196 


186.2 1 


10.95 


21 


255 


188.4 i 


10; 12 


22 


263 


190.89 


9; 60 


23 


237 


192.68 


7.68 


24 


211 


192.44 


4.44 


25 


218 


294.65 


2.90 


26 


210 


197. 19 


2.42 


27 


167 


198.50 


0.50 


28 


130 


20i .45 


-0.25 


29 


98 


202-07 


-2.66 


3b 


59 


205.64 


-2. 12 


3 j 


25 


208.96 


-2.88 


32 


12 


207.92 


-6.25 



STD. 
Dev. 

16;69 
13;60 
12.23 
14.95 
14.95 
15.37 
12.47 
13.70 
12; 54 
12.4 1 
i 1 .89 
12.70 
12. 16 
12.22 
9.69 
12.64 
il;83 
m.07 
id. 85 
9.92 
12.24 
I I .48 
7.22 
8.79 
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receive. In fact, the relationships with score gain that have been identified 
merely confirm the negative association with ability; gain is negatively 
related to ^^ariables generally associated with positive educational outcomes. 

For each UTK_ senior taking theCOMP exam in 1985 or 1986 an estimate of 
score gain w^s calculated in the following way: 

1) Ah estimated freshman COMP score was obtained from the ACT 
cbhcbrdahce table using the senior's entering ACT Assessment 
Composite score, then 



2) The estimated freshman COMP score was subtracted from the 
acuual senior COMP score. 

The derived gain scores were used in a series of one-way analyses of variance. 
Categorical variables included a variety of demographic characteristics, 
campus experiences^ and indicators of satisfaction. Separate analyses were 
conducted using the 1985 and 1986 data- Due to the large numbers of students 
involved—- 796 in 1985 and 2100 in 1986 ~ the -01 level of significance was 
used to identify variables associated with score gain (see Table 6 and 
Appendix B) - 

Factors found to be associated with the greatest mean gain in both years 
include : 

- High school grade point average less than 3-00 (B average) 

- Not receiving an academic scholarship 

- Father's education less than college graduate (Highest gain 

associated with less than high school education) 

- Non-participation in Honors English sections 

- Non-participation in Honors Math sections 

- Taking no more than two math courses 

- Taking either one or no social science course jor five or more such 

courses - 

Adding factors associated with gain in at least one of the two years, the 
evidence mounts that gain is negatively related to ability and to actions 
generally thought to contribute to student growth. In either 1985 or 1986 the 
greatest estimated COMP score gain was associated with the following: 

- College GPA less than 3.00 (B average) 

- Living Pt least a mile off campus and commuting to UTK 

- Working off campus 

- Working for at least 10 hours per week 

- Not even knowing about the orientation prcgram for freshmen 

- Using the Computing Center as little as possible 

- Taking as little natural science course work as possible 
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Table 6 

Analysis of Variance Results for Estimated COMP Gain 
(1985 and 1986_UTK_Seniors) 



Question 


1984- 


"85 


1985- 


-86 




dj. 


_F_ _ 


dr_ _ 


F 


oLUaenc s Kace 


1 T rt y- 

1 , 794 


0. 18 


1 i2098 


0.07 


Gendetr 


1 y 794 


1.16 


1 ,2098 


1 .01 


Marital Status 


1 , 794 


2; 01 


1 ,2098 


2.94 


Transfer Students 


1 , 798 


1.92 


1 ,2124 


9.15** 


High School GPA 


1 ,650 


14 . 74** 


1 ,2098 


0.07 


college GPA 


1,715 


4 .6 


1,1891 


51.11 


Type of Gommunity 


4 , 780 


0.88 


4 , 1871 


2.30 


Mother s Education 


5 ,782 


0.89 


5,1867 


2.77 


Father s Education 


5 ,778 


4 . 74 


5,1860 


5.08 


Parents * Income . 


8 , 764 


1 .69 


8 , 1825 


2.34 


Received Academic Scholarship 


1 ,788 


11 . 14** 


1 ,1878 


39.06 


Received Grant or Loan 


1 9 788 


2.81 


1 , 1871 


4 . 10 


Where the Student Lives 


4 i 780 


1 .66 


3^1872 


3.84 


Wnere the Student Works 


2 ,774 


7 .45** 


2,1853 


2.65 


Hours per Week — Working 


4 » 773 


4.63 


5,1852 


0.85 


Hours ^/Kit Week — Studying 


5 ,781 


2 . 22 


5,1871 


2.55 


Hours per Week — Library 


5 , 781 


3-36* 


5,1873 


5.13** 


Hours per Week — Pleasure Reading 


5,782 


0.93 


5,1874 


2.05 


Hours per Week — Television 


5,783 


d .86 


5,1873 


2.37 


Hours per Week — Social Activities 


5,782 


1 .93 






Hours per Week— Class 






5,1871 


1.17 


Commuter Student 


2,784 


_ _ 
4 .94* 






Participate in Freshman Orientation 




2 . 1871 


5.86 


Member Club or Professional Org. 


1,785 




0 . 58 


1 ,1876 


1-30 


Participate internship 


1,785 


2.53 


1,1877 


1 . 12 


Participate Co-op Program 


1,786 


0. 36 


1,1878 


6 .04 


Participate University, Studies 


1^785 


0. 30 


1,1879 


1.74 


Participate College_ Scholars 


1^785 


0.34 


1^1876 


1.77 


Participate Honors English 


1,784 


10.57 


1 , 1876 


30.32** 


Participate Honors Math 


1,784 


9.43* 


1*1876 


16.37** 


Campus Plays Attended 


4,785 


P .43 


4 » 1876 


2.89 


Campus Films Attended 


4,784 


1-33 


4 , 1875 


2.39 


Campus Concerts Attended 


4,785 


1 -79 


^ 9 1876 


1 .74 


Used Career Planning & Placement 


1,785 


4 .27 


1 , 1876 


Q-53 


Participate intramural Sports 


3,785 


1 .59 


3, 1877 


0.85 


Used Student Counseling Center 


3,784 


1 .90 


3, 1877 


1.29 


Used Student Employment Center 


3,783 


0.23 


3 1877 


0.75 


Used Computer Center 


3,785 


2.07 


3,1873 


6.82** 


Close Relationship with Faculty 


3,782 


3.46 


3, 1876 


1.48 


Hours per Week on Campus 


3,782 


2.92 


3, 1875 


2.29 


Foreign Language Coursework 


2,504 


4 .32 


2, 1248 


2.39 


Natural Science Coursework 


4,791 


2.30 


4,2092 


7.25** 


Mathematics Coursework 


4,786 


4 . 18* 


4,2077 


19 .02** 


Humanities Coursework 


6,783 


1.40 


6,2079 


1 .87 


Social/Applied Sciences Coursework 


6,788 


4 .06** 


6,2086 


6.70** 


History Coursework 


3,792 


1 .24 


3,207? 


0.56 


Satisfaction with Social Exper. 


3,783 


Q- 24 


3, 1870 


1.21 


Satisfaction with Academic Exper; 


3,783 


0 . /U 


3, 1875 


1 . 25 


Satisfaction with Overall Exper, 


3,783 


1.38 


3,1874 


1.07 


Expectation of Overall Experience 


2,783 


0.02 


2,1871 


0;28 


Offered Job Upon Graduation 


2,781 


2.79 


2,1859 


1.44 


Admitted Graduate/Prof. School 


2,780 


2.11 


2,1862 


4.08 


Plan to Remain in Tennessee 


2,783 


0.06 


2,1873 


4.43 



*A, detailed presentation of significant results is contained in Appendix B 
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Discussion 



Over the past three years a group of researchers at the University of 
Tennessee, Knoxviile have used the University ' s extensive database on the ACT 
eOMP Objective Test to investigate the concept of estimated score gain, or 
"value added" as those interested in accountability issues have called it; 
This work has led to the identification of five areas of concern that raise 
grave doubts about the reliability and validity of estimated score gain on the 
COMP exam. Three of the concerns are directly related to i. 'pliability; 
however, as an unreliable instrument or method is also invalid, these matters 
also have an important bearing on validity. Two of the concerns are clearly 
validity issues. 



Concerns Related ^o Reliability 

Work on three separate issues has pidduced strong evidence that estimated 
score gain does not have acceptable reliability. The first issue is a simple 
one of variability. Estimated COMP gain scores for 3,069 UTK seniors tested 
in two academic years were found to have such large standard deviations th ^t 
in a given year the estimate of gain for the senior class could be in error by 
as much as 26 percent. 

Since 1983 ACT staff have issued four concordance tables for use in 
estimating COHP_sco;e gain. Investigations undertaken at UTK have revealed 
significant differences among these tables^ and between estimates of score 
gain derived from the concordance tables and those derived from actual COMP 
exam scores earned by UTK freshmen^ In two recent years use of the ACT 

concordance tables to estimate COMP score gain for UTK seniors produced a 

level of gain 60 percent lower than that calculated from actual freshman COMP 
scores • 



Estimated gain is a change score — the difference between actual COMP 
score and an estimate of the freshman C^P score_based on the entering level 
of ability as measured by the ACT Assessmenc score — and thus :^'ts reliability 
is lower than the reliability of either of the measures on which it is based. 
Using measures of internal consistency and the intercorrelation of COMP scores 
achieved by UTK freshmen and seniors in the. calculation, reliability 
coefficients of .44 and .54 were obtained. Roughly half of the estimated gain 
score is attributable to error of measurement rather than true score. 

Concerns Related to Validity 

An :ristitutibh hoping to obtain ah indication of student growth in 
generic skills and knowledge fostered by its general education program must 
utilize a measure that is applicable to a valid sample of its undergraduates. 
For many senior institutions, no standardized measure of entering ability such 
as ACT or SAT score — the tiieasure upon which estimated COMP score gain is 
based --is available for a significant proportion of its graduates. At UTK 
one-third of the seniors do not have an ACT or SAT (which can be converted to 
act) score. Moreover, there are systematic differences between the group of 
seniors without such scores and the group having them. Seniors with no 
measure of entry level ability are more likely to be transfer students, older 
than average, from low-income families, and members of the black student 
population. No estimate of score gain based on entering ACT Assessment 
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eomposite scores can be considered valid for jprbgram evaluation at UTK because 
it is calculated using an invalid, i.e., unrepresentative ^ san>.ple of students. 

Qne^finai indication of the invalidity of estimated score gain on the 
COMP exam is provided_by a series of r^IationGhips with gain discovered in the 
course of the preliminary analyses conducted at UTK. One would hope that such 
research would suggest_directions for program impr^ivemeht that are consistent 
with those derived from conventional wisdom concer !ng good practice and the 
current literature in higher education. On the contrary, simple one-way 
analyses of variance provided no support for a number of programs and 
practices generally considered beneficial for college students. For example, 
estimated score gain is higher for students who have not participated in the' 
University's freshman orientation program, have not yet received an academic 
scholarship, have not ta':cen honors sections of English or math, have taken 
^^^^^f_^r no natural science and no more than two math courses, have seldom 
used the Computing Center, have resided at least a mile off campus^ and have 
worked off campus at least 10 hours per week. 



The nature of many of the foregdirig relationships is such, that this facet 
the research seems merely to confirm in a practical sense the statistical 
fioding that there is a significant negative correlation between student 
ability and estimated score gain on the COMP. At least in using the current 
list of some 50 variables available for association with estimated gain at 
UTKi .*:he relationships identified to date suggest no positive actions that 
could be taken by an institution to improve its general education program. 

Conclusion 

The ACT COMP exam has been demonstrated to be a valuable tool for 
stimulating faculty discussion ibout the general education currxculum, and 
°*9^^^_°^: i^^t^^ctidri, at colleges and universities across the country; The 
foregoing research does not detract from this substantial contribution to the 
improvement of the general education experience for the nation's college 
students . 

What is called into question is the usefulness, the validity^ of 
employing estimated student score gain on the COMP for the purpose of making 
preci3e judgments about program quality that can serve as the basis for 
decisions about the allocation of resources in higher education. Estimated 
score gain is just that ~ an estimate., it might be used in making r;ome gross 
distinctions between institutions that contribute to extraordinary student 
growth in generic knowledge and skills over four years and those that make 
little or ho contribution to such growth. But even this notion is merely 
speculative until confirmed through further research. To date we have only 
_go^ -hoe correlational studies i not the controlled experiments that are 
needed, to judge the efficacy of student score gain as an indicator of program 
quality. 

No institutioncan have a clear idea of the amount of student growth its 
general education program may be promoting until it tests its own incoming 
students, then administers an equivalent form of the same test to graduates. 
Longitudinal studies at a wide_variety of institutions with very different 
approaches to general education should shed some light on the validity of 
usins actual score gain in program evaluation. 
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It must be pointed but that there may be prbblems with the reliability 
arid validity of actual score gain as well._ Actual score gain is stili a 
change iscbfe, dependent on two measures of imperfect reliability. Freshmen 
who have taken the COMP exam will have had some practice that will influence 
their per fcrmance on the exam when they take it as seniors. (If all students 
^?y^__^^T SAT scores, thiis effect could be minimized in a cross-sectional 

calculated by teisting freshmen, and prepariug an 
inscitationai conc:ordancc table to serve as the basis for comparing scores of 
seniors of various ability levels.) Due to the ceiling effect , students of 
high ability who achieve high scores on the COMP as freshmen will not be able 
to show substantial growth in their scores_as seniors. Above all, there are 
enormous problems inherent in attempting to separate the effects on student 
development of a college curriculum as compared with all of the other life 
experiences students encounter during their years of association with the 
institution. However^ actual score gain will surely be an improvement over 
estimated gain id terms of its technical qualities. 

: . T^^ state of the art of measurement is sufficiently primitive for us to 

statement that we have just begun to scratch the surface in exploring 
of tests and measures for the purpose of assessing program 
?^?^9^i:^?"?^^^?_^^g^^^^^^^^tion. Appropriate caution must be exercised in 
^PP^?^?? iti decision-makihg before they are capable 

of providing information for making judgments of the pirecisibh required. 
Estimated score gain on the ACT COMP_exam is not a measure that can furnish 
data for precise decisions^ Additional research by many institutions is 
required to improve the quality of_ information that can be derived from score 
gain on the COMP. Such research will be costly and difficult, but it is 
imperative if educators and the public they serve are to be able to place 
confidence in score gain on the COMP exam as an indicator of program quality?. 
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APPENDIX A: 

CONTINGENT PROPORTIONS FOR STUDENTS WITH \ND WITHOUT ACT SCORES 

ON SELECTED VARIABLES 
(1984-85 and 1985-86 UTK Senior Samples) 
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STUDENT'S RACE (1984-85) 
i^hite N. White 
ACT 0.93 0.07 

Mon-ACT 0.88 0.12 
TOTAt 0.91 0.09 
)?=8.235; df=1| p<.01 

TRANSFER STUDENT (1984-85) 

Trans. N. Trans: 
ACT 0.14 0.86 

Non-ACT 0.63 0:37 
TOTAL 0.31 0.69 
XM48.8D6; df=i; p<.d01 

HIGH SCHOOL. G_PA (1984-85) 

<3.00 >3.00 
ACT 0.27 0.73 

Non-ACT 0.44 0.56 
TOTAL 0.30 0.70 
)?=17.701; df=1; p<.001 



STUDENT'S RACE. (1985-86) 
White N.White 
ACT 0.95 0.05 

Non-ACT g.75_ 0.25 
TOTAL __ 0.87 0.13 
X^293. 108; df=1; p<.0C1 



TRANSFER STUDENT (1985-86) 

Trans. N. Trans. 
ACT 0.17 0.83 

Non-ACT 0.52 0.48 
TOTAt 0.30 0.70 
X^469:832; df=1; d<.001 



HIGH SCHOOL 6PA (1985-86) 
<3.00 >3.S0 
0.31 0.69 
0.46 0.54 
0.34 0.66 
df=1; p<.001 



ACT 

Non-ACT 
TOTAL. 
X''=32.94; 



MOTHER'S EDUCATION (1984-85) 








< HS H.S: S:eol. 


Coi. 


S.Grad. 


Grad. 


ACT 0.85 0.32 0.34 


0.18 


0.04 


O.07 


Non-ACT 0.14 0.38 0.28 


0.12 


0.03 


0".D6 


TOTAL 0.08 0.34 0.31 


0.16 


0.03 


0.07 


X'^=45.49d; df=5; p<.001 








FATHER'S EDUCATION (1984-85) . 








< HS H.S.. S.CoL 


Col. 


S.Grad. 


Grad. 


ACT __ 0.06 0.20 0.22 


0.27 


0.05 


0.20 


Non-ACT 0.18 0.1? 0.23 


0.18 


0.03 


0.19 


TOTAL 0.11 0.20 0.22 


0.24 


0.04 


0.19 


X^=53.226; df=5; p<.001 








PARENTS' INCOME (1984-85) 








< 10K 10-20K 20-30K 


30-40K 


40-50K 


50-60K 


ACT 0.02 0.07 0.13 


0.17 


0.15 


0.10 


Non-ACT 0.D7 0.12 0.13 


0.14 


0.12 


0.10 


TOTAt 0.04 0.09 0.13 


0.16 


0.14 


0.10 


X^49.594; df=8; p<.001 









MOTHER'S EDUCATION (1985-86) 



< HS 

ACT 0.06 
Non-ACT 0.16 
TOTAL 0.10 
X*=98.5e9; df=5; 



H.S. 
0.29 
0.32 
0.30 
p<.0Ol 



S.Cbl. 
0.31 
0.26 
0.29 



FATHER'S EDUCATION (1985-86) 



ACT 

Non-ACT 
TOTAL 



< HS 
0.06 
0.17 
O.ld 



X'*=109.057; df=5; 



H.S. 
0.19 
3.24 

.0.21 
p<.0dl 



60-70K 
0.09 
0.05 
0.08 



70K+ 
0.13 
0.09 

o.n 



O.K. 
0.14 
0.18 
0.15 



S.COf. 
0.21 
0.18 
0.20 



Col. S.Grad, 

0.21 0.03 

0.16 0.03 

0.19 0.03 



Col. S.Grad. 
0.28 0.04 
3.19 0 " 
0.25 0 



Grad. 

o.p? 

0.07 
0.80 



Grad. 
0.22 



04 



0 



1? 
20 



PARENTS' INCOME (1985-86) 

:._ < IdK jO-20K 2d-3dK 30-40K 40-50K 50-60K 60-70K 70K+ D.K. 

ACT :__ 0.02 0.07 d.ll d.15 0.16 0.12 0.08 0.16 0.12 

Non-ACT p.D7 0.11 0.15 d.l4 0.11 0.09 0.06 0.11 0.17 

TOTAL 0.04 0.08 0. 12 0. 15 0. 14 d.lf 0.07 0. 14 G.U 
X*=95.682; df=8; p<.001 



SCHOLARSHIP STUDENT (1984-85) 



ACT 

Non-ACT 
TOTAL 



Yes 
0.28 
0.15 
0.23 



615; df=l; 



No 
0.72 
0.85 
0.77 
p<.001 



SCHOLARSHIP STUDENT (1985-86) 

Yes No 
ACT 0.29 0.71 

Non-ACT 0.18 0.82 
TOTAL 0.25 0:75 
X-^46.7d9; df=l p<.0O; 



EKLC 



WflEftE STUDENT LIVES (1984-85) _ WHERE STUDENT LIVES (1985-86) 





Dorm. 


Frat: 


Apart: 


Home 


Other 


Univ. 


Frat. 


< i mi > 


1 mi 




0:25 


0:05 


0;46 


0;21 


3; 04 


ACT 0.2§ 


0:05 


0.1S 


0:46 


Non-ACT 


0;17 


0.02 


0.48 


0.27 


0.06 


Non-ACT 0;2O 


O;02 


0;21 


0:56 


TGTAt 


0.22 


0.03 


0.47 


0.23 


0.05 


TOTAL G.26 


0.04 


0.20 


0.50 


X*=22.658; 


df=4; 


p<.001 








X-^55.424; df=3; 


p<.00l 







WHERE STUDENT WOk'KS (1985-86) 
N.^ork Camp, O.Camp^ 



ACT __ 0.45 


0.18 


0 


.37 












Non-ACT 0.42 


0.14 


0 


.44 












TOTAL 0.44 


0.17 


0 


39 












X*=18.823; df=2; 


D<.001 
















HOURS WORKiNQ (1985-86) 
















None 


1-9 


10- 


-1§ 


20- 


-29 


30 


-39 


40+ 


ACT 0.45 


0.07 


6 


21 


0 


18 


0 


0: 


0.03 


Non-ACT 0.41 


0.06 


0 


18 


0 


.17 


0 


.07 


0.11 


TOTAL 0.43 


0.07 


0 


20 


0 


18 


0 


06 


0.06 



XM7.733; df=5; p<.001 



HOURS SOCIAL ACTIVITIES (1984-85) 





None 


1-4 


5-9 


10-14 


15- 


19 


20+ 


ACT 


0:01 


0.18 


0.32 


0.25 


0: 


13 


0.11 


Non-ACT 


0.04 


0.28 


0.32 


8.20 


0: 


09 


0.07 


TOTAL 


0.02 


0.22 


0.32 


0.23 


0. 


11 


0.10 



)P=35.032; df=5; p<.0dl 

HOURS IN CLASS (1985-86) 







1-4 


5-9 


10 


-14 


15 


-19 


20 


-24 


25+ 


ACT 


0 


02 


0.07 


q 


24 


q 


43 


0 


15 


0.09 


Non-ACT 


P 


P_4 


0.13 


0 


.24 


0 


.37 


q 


15 


0.08 


TOTAL 


0 


03 


0.09 


0 


24 


0 


41 


0 


15 


0.09 



X'^38.320; df=5; d<.001 



COMMUTER STUDENT 


()984-f 


i5) 




Yes 


No 


D.K. 


ACT 


0.48 


0.51 


0.01 


Non-ACT 


0.58 


0.38 


0.04 


TOTAL 


0.52 


0.46 


0.02 


X*=23.371; 


df=2; 


p<.O01 





PART. FRESflMAN ORIENTATION (1985-86) 



Yes 




No 


D.K, 


ACT __ 0.7_5 


0 


16 


0.09 


Non-ACT 0.56 


q 


24 


0.20 


TOTAL 0.69 


0 


19 


0.12 


X*=123.704; df=2 


P< 


001 





MEMBER etU8/Ofi6ANiZATI0N (1 §85-86) 

Yes No 
ACT 0.51 0.49 

Non-ACT 0.40 0.6q 
TOTAL__ 0.47 0,53 
X*=28.591 ; df=1; p<:.001 
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HONORS ENGLISH (1984-85) 



Yes 


No 








ACT O.io 


0.90 








Non-ACT 0:03 


0.97 








TOTAt 0.08 


0.92 








X^24.0i9; df=1; 


D<:001 








CAMPUS PLAYS (1984-85) 








None 


1-4 


5-9 


10-14 


15+ 


ACT 0.23 


0.51 


0.18 


0.06 


0.03 


.Von-ACT 0.42 


0.36 


0.16 


0.04 


0.03 


TOTAL _ _ 0,30 


0,45 


0.17 


0.05 


0.03 


X-*^57. 127; df=4; 


f5<.001 








CAMPUS FILMS (1984-85) 




... : - 




None 


1-4 


5-9 


10-14 


15+ 


ACT 0.19 


0.36 


0.22 


0.12 


0:12 


Non-ACT 0.32 


3.35 


0.17 


0.06 


0.10 


TOTAL 0.24 


0.36 


0.20 


0.10 


0. i 1 


X^=39.361; df=4; 


p<.001 








CAMPUS CONCERTS (1984-85) 








None 


1-4 


5-9 


10-14 


15+ 


ACT 0.19 


0.55 


0.19 


0.04 


0.03 


Non-ACT 0.36 


g.4_7 


Q.13 


0.03 


0.01 


TOTAL___ 0,25 


0.52 


0.17 


0.04 


0.02 


J^=52.S07; df=4; 


p <.001 









HONORS ENGLISH- (1985-86) 
Yes No 
ACT 0.10 0.90 

Non-ACT 0.04 0.96 
TOTAL 0.08 0.92 
X'*=33.9d8; df=1; p<.0d1 



CAMPUS PLAYS (1985-86) 





None 


_:-4_ 




'0 


"14 


_15+ 


ACT __ 


0.25 


0.47 


0. 19 


p 


.05 


0.03 


Non-ACT 


0.41 


0.38 


0.13 


0 


.04 


0.33 


TOTAL 


0.31 


0.44 


0.17 


0 


.05 


0.03 


X-*^81.143; df=4; 


p<.001 










CAMPUS FILMS (1985-86) 












None 


1-4 


5-9 


13- 


-14 


:5+ 


ACT 


0.20 


0.38 


0.2] 


0 


.09 


0.12 


Non-ACT 


0.36 


0.33 


0.18 


0 


.06 


0.07 


TOTAL 


0.26 


0.36 


0.19 


0 


.08 


d.ic 




df=4; 


p<.doi 










CAMPUS CONCERTS (1984-85) 












None 


_1-4 


5-9 


10_- 


-14 


_15+ 


ACT __ 


0.23 


0.52 


0.18 


g 


.04 


0.03 


Non-ACT 


0.39 


0.43 


0.12 


0 


03 


0.02 


TOTAL 


0.29 


0.49 


0.16 


0 


.04 


0.03 



X-'=87.323; df=4; p<.O01 



CAREER PLANNING & PLACEMENT (1984-85) 

Yes No 
ACT 0.51 0.49 

Non-ACT 0.42 0.58 
TOTAL 0.48 0.52 
X*=9.074; df=l; p<.01 



CAREER PLANNING & PLACEMENT (1984-85) 

Yes No 
ACT 0.54 0.46 

Non-ACT 0:43 0.57 
TOTAL 0.51 0.4§ 
X*=32.299; df=4; o<.001 



INTRAMURAL SPORTS (1984-85) 

Never Seldom Occas. Freq. 

ACT 0.39 0.20 0.20 0.22 

Non-ACT 0.59 0.15 0.13 0.13 

TOTAL. _ 0,47 0,18 0.17 0.18 
X^=55.946; df=3; p<.D01 



INTRAMURAL SPORTS (1985-85) 

Never Seldom Occas. Frea. 

ACT _ 0.39 0.19 0.21 0.21 

Non-ACT 0.57 0.13 0.15 D.14 

.TOTAL___ 0,45 0,17 0.19 0.18 
X*^87.669; df=3; p<.001 



COUNSELING CENTER (1985-86) 



NiBver 


Sisldofn 


Occas. 


Freq. 


ACT 0.78 


0.16 


0.05 


0:01 


Non-ACT 0.76 


0.15 


0.06 


0.02 


TOTAL 0.78 


0.16 


0.05 


0.01 


X*=14.557; df=3; 


D<.01 






COMPUTER CENTER (1985-86) 






Never 


Seldom 


Occas. 


Freq. 


ACT 0.36 


0.20 


0.25 


0.20 


Non-ACT 0.44 


0.18 


0.20 


0.18 


TOTAL 0.39 


0.19 


0.23 


0.19 



X*ti8.182; df=3; p<.001 
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HOURS ON CAHPUS (1984-85) 




HOURS ON CAMPUS (1985-86) 














< 10 10-19 20-2? 


30+ 




< 10 


10-19 


20- 


-29 




30+ 






ACT 0,36 0,23 0,08 


0.33 


ACT 


0:38 


0; 19 


0 


;09 


a 


;34 






Nbn-ACT 0.49 0,23 0,09 


0.20 


Non-ACT 


0.48 




0 


;10 


0 


.21 






TOTAL 0.41 0,23 0,08 


0;28 


TOTAL 


3.42 


6.20 


0 


.09 


6 


.29 






X^=33.460; df=3; p<.001 




X^53.621 ; af=3; 


D<.081 


















NATURAL SCIENCE 


COURSEWORK 


(1985-86) 














3 






2 




3 




4+ 






ACT __ 


0.15 


0.49 


g 


.13 


n 

u 


.15 


0 


.08 






Npn-ACT 


0.17 


0.48 


0, 


.15 


n 


.14_ 


P 


.07 






_ TOTAL, _ 


0.16 


0,48 


0 


.14 


0 


.15 


0 


.08 






X'*=90.384; df=4; 


p<.001 


















MATHEMATICS COURSEMORK (1985-86) 
















0 


1 




2 




3 




4+ 






ACT 


0.12 


0.25 


0 


:10 




.31 


6 


.22 






Non-ACT 


0.14 


0.30 


0, 


90 


0, 


.24 


0; 


.22 






TOTAL 


0.13 


0.27 


0. 


.16 


6 


.29 


0 


.22 






)C*=188.346; df=4 


; p<.0O1 


















HUMANITIES COURStWORK (1985-85) 




















0 1 


2 


3 




4 




5 




6+ 






ACT 0.12 n.28 


0.22 


0.18 


p. 


,12 


0. 


.06 


P, 


.P2 






Non-ACT 0.12 0.31 


0.19 


0.17 


0. 


11 


0. 


07 


0. 


03 






TOTAL 0.12 0.29 


0.21 


0.18 


0. 


12 


0. 


,06 


0. 


,02 






X''='!ia.984; df=6; p<.001 






















SCCIAL/APPLieO SCIENCES 


eOURSEWORK (1985-81 


5) 
















0 ; 


2 


3 




4 




5 




6+ 






ACT 0:07 0:12 


0.12 


0.18 


0. 


23 


0. 


18 


0. 


10 






^on-ACT 3.87 3.15 


0.13 


0.22 


0. 


21 


0. 


13 


0. 


08 






TOTAL 0.07 C.13 


o.:2 




G. 


22 


0. 


16 


0. 


09 






)(*=158.239; df=5; D<.001 


















HISTORY COURSEWORK (1984-85 J 






















q _ .1 _ _2_ 


_ 3+ 




















ACT __ 0.43 0.4? 0.p_7 


g.gi 




















Non-ACT 0.24 0.65 0.09 


0.01 





















TOTAL 

X*=49.536; df=3; p<.001 



SATISFACTION SOCIAL EXPERIENCE (1984-85) 



V.Sat. 

ACT 0:40 0.46 

Non-ACT 0:28 0.55 

TOTAL 0.35 0.50 

X^18.232; df=3; p<.0Dl 



Sat. Dfssat. 

0:11 
0.13 
0.12 



V;Dis. 
0.03 
0.03 
0.03 



SATISFACTION ACADEMIC EXPERIENCE (1?84-85) 



V.Sat. 
ACT 0.22 
Non-ACT 0.28 
TOTAL 0.24 



Sat. Oissat. 
0.53 0.12 
0.53 0.15 
0.60 0.14 



V.Ois. 
0.02 
0.03 
0.03 



X*=13.105; df=3; p<.01 



SATISFACTION SOCIAL EXPERIENCE (1985-86) 



V.Sat. 
ACT 0.40 
Non-ACT 0.31 
TOTAL P..37 
X'*=30.779; df=3; 



Sac. Dissat. 



0.45 
0.51 
0.48 

p<.O0i 



0.11 
0.14 
0.12 



V.Ois. 
0.03 
0.05 
0.04 
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SATISFACTION OVERALL EXPERIENCE (1984-85) 



SATISFACTION OVERALL EXPERIENCE {1985-86) 



V:Sat; 


Sat. 


Dfssat. 


V:Dfs; 


V.Sat. 


Sat. 


Oissat. V.Ois. 


ACT 0:35 


0;55 


0;09 


0:01 


ACT 0:33 


0.55 


0.10 0.02 


Nbn-ACT 0.28 


0.58 


0.12 


0.02 


;ion-ACT 0;29 


0;54 


0:14 0.03 


TOTAL 0.32 


0.56 


0.10 


0.02 


TOTAL 0.32 


0.55 


0;12 0;02 


X*=12.288; df=3; 


p<.01 






X*=15.703; df=3; 


D<.00] 












PLAN TO REMAIN IN TENNESSEE (1 9/15-86) 










Yes 


No 


O.K. 










ACT __ 0.39 


0.26 


0.35 










Npn-ACT 0.46 


0.25 


0.29 










TOTAL 0.42 


0.26 


0.33 










X'^12.430; df=2; 


p<.01 
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APPENDIX B: 
61EAN GAIN SCORES FOR SEtECTEfi VARIABLES 
(1984-85 and 1985-86 UTK Senior Samples) 



32 



GAIN GAIN 

QUESTION 1984-85 1985-86 

High School Grade Point Average 

Less Than 3.00 12.31 11.86 

3.00 or Greater 8.00 6.61 

College Grade Point Average 

Less than 3.00 9.97 

3.00 or Greater 5.33 

Father's Education 

Less Than H.S. 12.60 11.17 

Graduate ? . 99 8 . 88 

Some College 11.70 9.85 

College Graduate 6.46 7.31 

Some Grad. School 8.24 5.10 

Graduate Degree 7.61 6.24 

Received Academic Scholarship 

yes 6.64 4.96 

No 9.98 9.38 

Where Student Lives 

yniverslty Housing 6.67 

Fraternity 6.64 

Less Than 1 *Mile Away 7.66 

kovB than l Mills Away 9.15 

Where Student Works 

Not Workincr 7.04 

Working On Campus 9.01 

Working Off Canptas 10.90. 

Hours Per Week - Working 

None 7.09 

1 - 9 7.44 

10 - 19 10.21 

20 - 29 11.39 

30+ 11.51 

Hours Per Week - Library Materials 

None 5.92 

i - 2 8.92 

3-5 8.08 

6-8 11.30 

9 - 11 9.76 

12+ 3.32 
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QUESTION 



GAfN 
1984-85 



GAIN 
1985-86 



Commuter Student 
Yes 
No 

Not eertain 

Freshman brlentation 
Yes 
No 

Didn't know About 

Honors English 
Yes 
No 

Honors Math 
Yes 
No 

Use Computer Center 
Wever 
Seldom 

Occasionally 
Frequently 

Natural Science Coursework 
None 
1 
2 
3 

4^ 

Mathematics Cdursewbrk 
None 
1 
2 
3 

4+ 



10.50 
7.67 
7:22 



4.69 
9.57 



-3.10 
9.20 



8.97 
11.27 
10.01 
8.98 
6; 02 



7 . 69 
8.55 
11.63 



2.76 
8.68 



0.83 
8.33 



9.28 
?. 14 
7.44 
5.51 



10.04 
9.23 
8.85 
5.38 
6.19 



12.03 
9.33 
8. 15 
9.5? 
4.07 
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QUESTION 



Social Science Coursewdrk 
None 

i 

2 
3 
4 
5 

6 + 



GAIN__ 
1984-85 



3.40 
10.42 
7.57 
6.77 
7.91 
11 .78 
11 .76 



GAIN 
1985-86 



11 .01 
6.41 
6.40 
6.73 
8.11 
11 .08 
10.83 
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